A lagrange programming neural network approach for nuclear norm optimization

This article proposes a continuous-time optimization approch instead of tranditional optimiztion methods to address the nuclear norm minimization (NNM) problem. Refomulating the NNM into a matrix form, we propose a Lagrangian programming neural network (LPNN) to solve the NNM. Moreover, the convergence condtions of LPNN are presented by the Lyapunov method. Convergence experiments are presented to demonstrate the convergence of LPNN. Compared with tranditional algorithms of NNM, the proposed algorithm outperforms in terms of image recovery.


Introduction
Recently, many researchers were increasingly interested in collecting useful messages from very limited information.Most real data matrices restoring information are low-rank or approximate low-rank.In other words, a fundamental assumption of the problem is that the target matrix has a low-rank structure.Thus, we call similar issues of a low-rank structure to be the Rank Minimization Problem (RMP).Some applications (e.g.control [1], machine learning [2], and computer vision [3]) can be formulated as different versions of RMPs.In the view of applications, RMP can be commonly reformulated as a problem of reconstructing data matrix from partial matrix sampling.Suppose that M 2 R n1�n2 is a target matrix with partial observations.O is a set of known elements in matrix M. X is a low-rank matrix after recovered and rank(X) is the rank of X. RMP can be formulated into the following optimization problem min rankðXÞ s:t: Due to the nonconvexity and combinatorial nature properties of rank functions, problem (1) is an NP-hard problem.Candès et al. [4] proved that the majority missing elements of a matrix with the rank r can be properly restored with a high probability by using a simple convex optimization programme in case of n ¼ maxðn1; n2Þ; m � Cn 1:2 r log n; where C is a certain positive numerical constant.Commonly, the above-mentioned problem can be reformulated as a nuclear norm minimization problem (NNM).The task of fully recovering the majority missing elements M with rank r can be accomplished by NNM.Problem (1) can be rewritten as where kXk * ¼ P k i¼1 s i ðXÞ is the nuclear norm of X, σ(X) is the singular value of the matrix X and k = min(n1, n2).
Neural networks were used to address nonlinear equality constraint problems.Many researchers aimed to reformulate the optimization problem (3) and solve it by neural networks [13,14].The main advantage of neural networks is to generate real-time solutions.Zhang and Constantinides [15] firstly created a Lagrange programming neural network (LPNN) to address a variety of nonlinear constrained optimization problems.After that, the global convergence conditions of LPNN were attracted the attention [16,17].Recently, a large number of variants of LPNNs were proposed to address different application problems.Feng et al. [18] proposed a new LPNN approach to recovere sparse signals based on the locally competitive algorithm (LCA).Liang et al. [19] and Shi et al. [20] applied LPNN to address the radar location problem.Xiong et al. [21,22] applied LPNN to Robust TDOA source localization problem and Elliptic target positioning problem.Based on above mentioned researches, we try to solve problem (3) by using LPNN.The contributions are as follows: • The nuclear norm minimization problem (NNM) is reformulated as a optimization problem of the matrix form, and we solve NNM by using a Lagrange programming neural network method (NNM-LPNN).
• By constructing an appropriate Lyapunov function, we propose the convergence conditions to prove the stability of NNM-LPNN.

Notations
Suppose that R m�n is m × n-dimensional real matrices.The capital A is a matrix and a (i, j) is the the i, j-th entry of A. A > , rank(A) and Tr(A) are the transposition, the rank and the trace of A, respectively.kAk F and kAk * represent the Frobenius norm and the nuclear norm of A. ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi TrðA > AÞ p ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi P i;j a 2 ij q , and kAk r max is the largest singular value of A. hA, Bi F and A � B are the Frobenius inner product and the Hadamard product of A and B.

Subdifferential
The definition of subdifferential is put forward in [23].Definition 1.Let the function f : O m�n !R be convex and O be convex.For any D 2 R m�n , we have so that Δ 2 @f(X), where @f(X) serves as the subdifferential of f(X).The partial differential function of f(X, Y) with regard to X is denoted by the symbol r X f(X, Y).
Proof.Let X, Y 2 R m×n , according to Definition 1, we have Add (5a) and (5b), we obtain A matrix X can be precisely represented by its singular value decomposition (SVD) where the elements ρ 1 , ρ 2 , . .., ρ r � 0 are the singular values, the singular vectors are u 1 , u 2 , . .., u r 2 R m and v 1 , v 2 , . .., v r 2 R n .In [24], Y has the form In this article, the subgradient of the nuclear norm is represented as follows: Remark 1.The computation of the subgradient of the nuclear norm can be derived using the formula in (8).In [24], minimum-norm subgradient can be used to ensure convergence.Since E is SVD decomposition from X, it is only necessary to minimize Y by minimizing kWk F .Due to the positive definiteness of Frobenius norm, we can easily get W = 0.

Lagrange programming neural network
Problems involving general nonlinear programming with equality requirements can be solved using the following LPNN method min X f ðXÞ; s:t: hðXÞ ¼ 0 ð10Þ where f ðXÞ : R m�n !R is the objective function, hðXÞ : R m�n !R m�n describes the m × n constraint on equality.Suppose that both functions f and h are quadratically differentiable.A Lagrangian function of problem ( 10) is constructed as follows: Λ is a matrix form of Lagrange multipliers.In the view of the structure of neural networks, variable and Lagrange neurons cannot belong to the similar types of neurons.In other words, the entries of X and Λ are stored in variable neurons and Lagrange neurons, respectively.The dynamics of neurons determined by where τ 0 is the circuit's time constant.
3 Lagrange programming neural network for the nuclear norm optimization

NNM properties
NNM can be formulated as According to problem (13), we have the following proposition.Proposition 1.Let X ?an optimal solution of problem (13) if and only if there exists a Λ ? and the following conditions satisfied.
The conditions for the Karush-Kuhn-Tucker (KKT) equation are outlined in Proposition 1 by ( 14) in its entirety.The KKT requirements are both necessary and sufficient because the problem is convex.

NNM-LPNN dynamics
To avoid stability issues around equilibrium, we provide an enhanced term 1  2 kW � ðX À MÞk 2 F in the Lagrange function as follows: The objective value at an equilibrium X* is unaffected by the addition of the augmented term, because W � (X* − M) = 0.The gradients of L nnm are given by where Y 2 @ k Xk * .
A neurodynamic centralized method for solving(13) is Theorem 1. Suppose that {X ?, Λ ?} serves as the LPNN dynamics of the equilibrium point of system (17).The KKT criterion of problem (15) is satisfied at the equilibrium point.Thus, the KKT criterion of NNM is necessary and sufficient.The stabilization point of (17) is consistent with the desired solution of problem (15).
Proof.By the definition of stabilization points Based on ( 17) and ( 18), we can conclude that Then, we have With Y ? 2 @ k X ?k * , we have Therefore, (16a) is also satisfied.Similarly, we can prove that (14b) leads to (19b).The proof is finished.Theorem 2. The neurodynamic method (17) globally converges to the optimal solution of the problem (13).
Proof.Construct a Lyapunov function: By definition, the above function V 1 (X, Λ) is semi-positive definite and it is radially unbounded in the domain of definition.The derivative of V 1 can be expressed as From ( 17) and ( 19), we have Substituting ( 24) into ( 23), we obtain From Lemma 1 and the definition of W, we could get According to (26) and k _ Lk In other word, for arbitrary initial value fX; Lg 2 R m�n , V 1 (X, Λ) is nonincreasing when t !+1.Hence, the proof is completed.

Experiments
We carry out experimental simulations of the following aspects to verify the performance of the proposed approach (NNM-LPNN in short): 1. Optimality and convergence.
3. Recovery ability of low-rank images at different sampling rates (SRs) and comparison with classical Singular Value Thresholding (SVT) approach and alternating direction multiplier method (ADMM) [25] and fixed-point continuation(FPC) [9].

Low-rank numerical matrix recovery
Example 1: M 2 R m�n is synthesized obeying distributed standard Gaussian distribution [4].Giving a rank r, we generate two matrices M m×r and M r×n using NNM-LPNN.Suppose that m = n = 50 and r = 5.The matrix reconstruction result is shown in

Conclusion
In this article, a Lagrange programming neural network was proposed to solve the nuclear norm minimization.The stability and optimality of the proposed approaches were proved theoretically.Low-rank matrix recovery and approximate low-rank image recovery experiments demonstrated that the presented neural network is effective.

Fig 1 .
The recovery results with SR = 50% are shown.This phenomenon indicates that NNM-LPNN can recover the target M accurately.Moreover, the optimal solution of problem (13) is shown in Fig 2.Its reconstruction error and relative error are shown in Figs3 and 4. In summary, the NNM-LPNN can reach the equilibrium point and achieve the optimal solution.

Example 2 :
We select a real image as shown in Fig 5 and its first 50 singular values are shown in Fig 5.This image can be considered to be a low-rank image.As shown in Fig 7, we sampled the image at SR = {10%, 50%, 90%} and the reconstruction results of the three approaches are